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ABSTRACT 



This paper reports the results of observation skills 
training via an interactive multimedia training program for preservice 
teachers at Lincoln University (Missouri) . The training program provides 
multiple videos for users to practice six types of observation methodologies, 
comparisons to "expert" data, and on-line access to procedural information. 
The study provides an analysis of audit trail records, user surveys, and 
pre-post testing of observer reliability to ascertain relationships between 
computer experience, prior classroom experience, learning strategies, 
engagement time, and program efficiency. Sample screens from the multimedia 
training program are included. Four tables present data on: observation 
reliability score outcomes for two observational methodologies; relationships 
between outcomes and learner differences; relationships between outcomes, 
learning style differences, and semester; and relationships between outcomes 
and practice strategy differences. (Author/DLS) 
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Abstract: This paper reports the results of observation skills training via an interactive 
multimedia training program for preservice teachers. The training program provides 
multiple videos for users to practice six types of observation methodologies, comparisons to 
"expert" data, and on-line access to procedural information. The study provides an analysis 
of audit trail records, user surveys, and pre/post testing of observer reliability to ascertain 
relationships between prior computer experience, prior classroom experience, learning 
strategies, engagement time, and program efficiency. 

1 Background to the Study 

The purpose of this study was to examine the effectiveness of a multimedia training program to 
develop proficiency in classroom observation skills of preservice teachers. The training program was 
delivered through an interactive videodisc format which offers multiple videos for teachers to practice six 
types of observational methodologies: event recording, duration recording, latency recording, interval 
recording, time sampling, and A-B-C analysis [Fitzgerald 96] [Semrau 96]. For each video-based practice, 
the computer program controlled the start and stop frames, provided a timer, and provided user control over 
practice sequence. Users entered their observation results into the computer program and compared their data 
to that of "experts." Two unique features in the design of the multimedia training program are: 1) one video is 
"saved" to provide a constructivist experience for the user to define behaviors for each of the observational 
methodologies, and 2) procedural information for the six observational methodologies is available on-line for 
easy review. 

Training in classroom observation skills is a complex and time-consuming process. To learn to code behavior 
reliably, an observer must learn to identify behaviors accurately, watch and code multiple behaviors in real- 
time, and apply these skills to varied and dissimilar settings. The usual training method involves an 
apprenticeship model where a teacher explains the procedure, demonstrates via videotapes, provides practices 
through videotapes and field assignments, and undertakes reliability testing with tapes, if at all. It is not 
feasible to spend the necessary amount of practice time for classroom observation training within most 
teacher education programs; thus mastery of methodology or reliability is rarely documented [Fitzgerald, 
1995]. 

Interactive multimedia training provides one solution to the problem of observation training, practice, and 
reliability testing. Through repeated practice with videos and "expert" comparisons, a novice can practice 
until proficient and establish coding reliability. With the capability of accessing procedural information 
C& regarding observation methodologies, a novice can review procedures if questions arise during practice. Since 

(A a hypermedia format allows users to make personalized decisions regarding practice sequences and strategies, 



the format may provide an effective training alternative to the typical apprenticeship model in observation 
training. 

2 Description of the Multimedia Training Program 



This screen introduces the user to the 
particular observational methodology to be 
practiced. In this example, the user is practicing 
Event Recording. A brief description of the 
methodology provides general information 
necessary to begin the practice. If the user 
requires further information, he/she may access 
procedural information from the on-line resource 
section of the program. A sample table gives the 
user an example of how event data are recorded. 
The user chooses from three possible videos for 
practice, two which include expert observation 
results and one that does not. The observation 
practice M Do Your Own” can be used as a 
constructivist activity for defining behaviors of 
interest to the user. 

After the user runs the video and collects 
his/her observation data, the data entry screen 
appears. At this step, the user can elect to run the 
video again or enter the data which he/she has 
collected into the program. The computer screen 
provides the exact length of the observation 
period to enable the user to compute rate data for 
the observed behavior. These data entry screens 
are matched to each type of observation 
methodology which vary in the number of 
children and behaviors observed and the method 
for computing rate data. 





After the user enters his/her data into the 
computer, the expert comparison data screen 
provides comparative results. The screen displays 
both the user's results and the results of experts 
who observed the same observation video. 
Information at the top of the screen explains how 
the user can compute his/her "percentage of 
agreement" (reliability) between the two sets of 
data. At this point, the user may go back and 
practice this observation segment again, select 
another segment to practice, or return to the main 
menu and select a new methodology for practice. 

Audit Trail 




The program records embedded data paths that are invisible to the user. These paths collect information such 
as the overall length of time the user engages in the program, the length of time spent in each methodological 




section, the number of times a video is accessed, and the observation results entered by the user for each 
practice. For example, in the above practice, the computer records the frequency and rate data entered by the 
user for that particular practice semester. The audit trail provides a cumulative record of all practices and 
observational results and enables the researcher to examine factors which may relate to successful learning. 

3 Description of the Research Study 

3.1 Research Questions 

The study was designed to address the following research questions about the use of hypermedia-design 
practice materials for learning classroom observation skills: 

(1) What effect does a hypermedia-designed practice program have on performance outcomes for 
preservice teachers learning classroom observation skills? 

(2) Do performance outcomes for preservice teachers relate to differences in prior computer 
experience, 

prior hypermedia experience, prior teaching experience, learning style preferences, or semester? 

(3) What practice strategies predict successful performance outcomes for preservice teachers 
learning 

classroom observation skills through a hypermedia-designed practice program? 

3.2 Methodology for Implementing the Multimedia Training Program 

The observation training program was integrated into special education methods courses during three 
semesters at Lincoln University. Following pre-testing, the instructor provided instruction in observation 
skills prior to the students' use of the multimedia program. The instruction included content and procedural 
information and a demonstration of each methodology. During these introductory sessions, students could ask 
questions about the observation procedure and clarify the required observation steps. Students were then 
required to practice the observation methodologies using the multimedia training program in the computer 
lab. The basic expectation was that each student use all available practices with the suggestion that practices 
be repeated if the user's data did not match the expert data by 80% agreement. Students received extra credit 
for their participation, but their level of proficiency was not used in computing the course grade. 

3.3 Participants 

Subjects included 29 preservice teachers enrolled in special education methods courses. All students 
were undertaking foundational coursework prior to student teaching; students in the summer session had prior 
knowledge of special education methodology; fall and spring session students had little prior knowledge. 
Demographic information was collected via user surveys and served as independent variables. 

Prior Computer Experience. All but two participants had a moderate level of experience using 
computers across a number of areas of computer applications; the remaining two had minimal experience. 
Based on a scale where students rated their experience from 0 = "no knowledge" to 9 = "expert knowledge" 
on 9 categories of computer applications (total score possible = 81), the mean prior computer experience 
score = 29.7, standard deviation = 14.6 , range = 2 to 53. Most students had never experienced hypermedia, 
and those that had, indicated they had limited knowledge of hypermedia. Using the sub-scale for rating 
experience with hypermedia (item score possible = 9), the mean prior hypermedia experience score = 0.9, 
standard deviation = 1.4, range = 0 to 5. 

Prior Teaching Experience. Since the subjects were enrolled in an initial teaching certification 
program, few had direct, salaried, classroom teaching experience. Of the seven with teaching experience, they 
had been primarily employed as teacher aides or substitute teachers. The mean years of prior teaching = .88, 
standard deviation = 2.03, and range = 0 to 7 years. 
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Learning Styles. Participants represented a mix of learning styles as assessed with the Kolb Learning 
Style Inventory (McBer & Company, 1981). Using the 4 learning style types defined on this instrument, there 
were 9 students identified as assimilators, 9 as divergers, 8 as accommodators, and 3 as convergers. 

Semester. Participants were included in the study over a three-semester duration. The numbers of 
students for the semesters are: summer = 7, fall = 16, and spring = 6. There were some implementation 
differences across the semesters particularly affecting the duration of practice time prior to post-testing. 

4 Data Collection for the Dependent Variable 

The dependent variable for the study was observation proficiency measured by pre- and post-testing 
on two different types of observation methodologies — event recording and interval recording. Matched pre- 
and post-tests were administered using original videos controlled by the computer to standardize test 
administration. In the event recording test, subjects watched 5-minute videos and recorded the frequency of 
occurrence of a defined behavior. In the interval recording test, subjects watched 5-minute videos and 
recorded on-task in each of 50 six-second intervals. Percentage of agreement scores (reliability) were 
computed for each of the pre- and post-tests. 

Path analysis data were compiled from the on-line audit trails and user practice logs. Through these 
logs, practice strategies were examined by analyzing each users number of repeated practices, total practice 
time in the program, number of times the procedural information module was accessed, total time spent 
reviewing procedural information, and proficiency on two related observation practices: interval practice # 2 
and time sampling practice #2. These two internal practices were used to ascertain proficiency of users during 
practice. 

5 Data Analysis, Results, and Interpretation 

5.1 Question 1 : Performance Outcomes 

A paired t-test was applied to the pre- and post-test results to determine the effect of the hypermedia- 
design practice program on observation skill proficiency. The scores represented the percentage of 
agreement (reliability) to expert data. Pre-to-post test results were not significant for event recording 
(t (24) = .959; p = .35) but were found to be significant for the interval recording methodology ( t 
(24) = 3.88; p =.001). Table 1 displays the means and standard deviations for the reliability scores. 

The small variation found in pre- and post-scores for both event and interval tests suggests the 
difficulty level of these tests is not high enough to provide accurate assessment of observational 
ability by the participants in these areas. In addition, the narrow range of scores exhibited limits the 
effectiveness of the regression analysis. In the future, more robust pre/post measures might ensure 
greater assessment of proficiency. 

Further recommendations include increasing the number of practices available to users within the 
program. The ability to check one’s scores with the results of the ’’experts" is an integral and 
essential part of the program; however, once these results are seen, repeat practices are 
compromised, essentially allowing for only two ’’unique” practices in each observational method. 
Additional practices would increase the amount of unique experiences available. 





Pre-test Reliability 


Post-test Reliability 




Mean 


S.D. 


Mean 


S.D. 


Event Recording 


79.74 


20.16 


85.78 


14.14 


Interval Recording 


78.98 


7.02 


84.32 


5.06 



Table 1. Observation Reliability Score Outcomes for Two Observational Methodologies 
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Question 2: Relationships between Post-test Outcomes and Learner Differences 



Simple regressions were run to evaluate whether outcomes of the observation training program were 
related to learner differences. Table 2 presents the regression results for the post-test outcome scores 
related to three independent variables. No significant relationships were found to indicate that prior 
computer experience, prior hypermedia experience, or prior teaching experience with children 
predicted the outcomes for learning observation skills. 



Post Event Recording 
r F- value p 


Post Interval Recording 
r F-value p 


Prior Computer Experience 


.115 


.293 


.59 


.088 


.173 


.68 


Prior Hypermedia Experience 


.195 


.271 


.36 


.112 


.281 


.60 


Prior Direct Teaching Experience 


.241 


1.361 


.26 


.176 


.704 


.41 



Table 2. Relationships between Outcomes and Learner Differences 

To evaluate whether outcomes were related to learning styles or differences across the three 
semesters, ANOVAs were run learning style and semester as factors. As shown in Table 3, no 
significant differences in learning outcomes were related to learning styles of the participants or to 
semester. 







Post Event Recording 


Post Interval Recording 




n 


Mean 


F-value 


P 


Mean 


F-value p 


Learning Stvle Tvoe 
Assimilator 


8 


87.94 


1.303 


.30 


84.75 


.443 .72 


Diverger 


8 


79.69 






83.50 




Accommodator 


7 


88.50 






83.71 




Converger 


2 


100.00 






88.00 




Semester 

Summer 


7 


92.86 


1.78 


.19 


86.57 


1.37 .27 


Fall 


12 


80.79 






84.17 




Spring 


6 


87.50 






82.00 





Table 3. Relationships between Outcomes, Learning Style Differences, and Semester 

The subject pool for this study exhibited a high level of homogeneity in prior computer experience, 
prior hypermedia experience, and prior teaching experience. As a whole, most subjects had a 
moderate amount of computer experience, minimal hypermedia experience, and little, if any prior 
teaching experience. This limited range of experiences negatively impacts the effectiveness of the 
regression analysis. The small number of subjects also limits accurate evaluation, particularly with 
regard to subjects' learning styles. Further studies, therefore, would greatly benefit from a larger and 
more diverse subject pool. 

5.3 Question 3: Relationships between Outcomes and Practice Strategies 

Path analysis data were used to examine relationships between successful learning outcomes on the 
interval post-test and strategies used during practice on the multimedia program. Simple regressions 
were run to evaluate whether outcomes of the observation training program were related to 
differences among the subjects in how they utilized the program capabilities, amount of practice 
time, and proficiency levels during practice on the more rigorous methodologies. Table 4 presents 



6 



the regression results pertaining to audit trail data. No significant relationships were found to indicate 
that different ways subjects utilized the program affected their performance outcomes. 

The number of practices and amount of total time spent in the program exhibited a better distribution 
among the subject group. However, both were found to be insignificant to successful learning 
outcomes. 

The available review section in the hypermedia program was largely unused by the subjects because 
of the amount of procedural knowledge provided by the instructor. This reduced the importance of 
such a knowledge source in the hypermedia program. Therefore, it is recommended that hypermedia 
developers consider the environment in which the program is to be used and determine the necessity 
of such a 



Post Interval Recording 





r 


F -value 


P 


Number of Repeated Practices 


.233 


.974 


.34 


Total Practice Time 


.091 


.143 


.71 


Number of Accesses to Procedural Information 


.082 


.155 


.70 


Total Time Accessing Procedural Information 


.028 


.018 


.89 


Proficiency on Interval #2 Practice 


.226 


1.243 


.28 


Proficiency on Time Sampling #2 Practice 


.057 


.055 


.82 



Table 4. Relationships between Outcomes and Practice Strategy Differences 



6 Summary 

Interesting questions emerge regarding the effectiveness of an interactive multimedia training 
program for developing observation skills. Because the program allows users nonlinear access to practice 
segments, expert comparisons, and access to procedural resource information; the learning strategies of users 
differ. The findings from this study provide multiple data points from audit trail records, user surveys, and 
pre/post testing in observation skills to examine the correlates of observation proficiency: prior computer 
experience, prior classroom experience, learning strategies during training, and total engagement time with 
the program. 

Students utilizing this program appeared to learn observation skills equally well, regardless of 
different learning styles or their prior experiences with computers or in the classroom. Given the limiting 
factors of a small and homogeneous subject pool, this study nevertheless suggests that a hypermedia-based 
practice environment minimizes the effect of learning style and experience differences between users in terms 
of overall outcome of observation proficiency. This study therefore lays the groundwork necessary for further 
study of the equalizing potential of hypermedia-based learning. 

The study is important in establishing how preservice teachers utilize an interactive multimedia 
learning environment for observation skills training, the resulting outcomes for observation proficiency, and 
overall training efficiency of the multimedia program. 
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